Skip to content

fix(writer): the child nodes of a flat segment carry the array stats Rust keeps (#583) - #674

Merged
dfa1 merged 1 commit into
mainfrom
feat/583-child-node-stats
Nov 6, 2026
Merged

dfa1 merged 1 commit into
mainfrom
feat/583-child-node-stats

Conversation

@dfa1

@dfa1 dfa1 commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Part 2 of #583 (after #673).

The cascade attaches Rust's stats to every child it encodes, not just the root, plus sequence sorted flags, patch-values is_constant=false, and the validity bitmap stats (is_constant, or min/max/sum/null_count). NodeStatsInteropIntegrationTest compares the whole tree of the numeric columns against vortex-jni.

Not covered yet: string children (FSST/onpair values depend on the symbol table) and the zone-table segment shape of string and bool columns (PR 3).

Refs #583

🤖 Generated with Claude Code

https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV

…Rust keeps (#583)

Rust's compressor leaves its cached stats on every array it compresses, so
the children a scheme cascades into carry them as the root does. The cascade
now attaches the same stats to each child it encodes (RootNodeStats moves to
writer.encode), and three shapes of their own:

- vortex.sequence writes is_sorted / is_strict_sorted (Sequence::stats: false
  for a falling step, true for fewer than two values)
- the primitive values of ALP and ALP-RD patches carry is_constant = false
- the vortex.bool validity bitmap of a nullable primitive is is_constant when
  every row has one value, otherwise min, max, sum (valid rows) and
  null_count = 0; NodeStats gains the sum component

NodeStatsInteropIntegrationTest now compares the whole tree of the numeric
columns against vortex-jni, with patched-float and falling-sequence columns.
String children (FSST/onpair, which depend on the trained symbol table) and
the zone-table segments of string and bool columns remain.

Refs #583

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV
@dfa1
dfa1 merged commit e2ec9b5 into main Nov 6, 2026
6 checks passed
dfa1 added a commit that referenced this pull request Nov 6, 2026
… with unit tests (#583)

Review of #674: RootNodeStats is now the package-private CachedNodeStats, reached from
VortexWriter through one public EncodeResult#withRootStats. Unit tests cover the cached stats on
dict children, the validity bitmaps of a nullable integer and float, patch values and the
fill-only-when-empty rule; the interop test gains a nullable f64 column. Docs: three shapes, and
bit-packed patch values.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV
@dfa1
dfa1 deleted the feat/583-child-node-stats branch November 18, 2026 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant